Machine learning based assignment of service levels in a networked storage system

ABSTRACT

Methods and systems for a networked storage system is provided. One method includes transforming by a processor, performance parameters associated with storage volumes of a storage system for representing each storage volume as a data point in a parametric space; generating by the processor, a plurality of bins in the parametric space using the transformed performance parameters; adjusting by the processor, bin boundaries for the plurality of bins for defining a plurality of service levels for the storage system based on the performance parameters; and using the defined plurality of service levels for operating the storage system.

CROSS-REFERENCE TO RELATED APPLICATION

This patent application claims priority under 35 USC § 119(e) to U.S.Provisional Patent Application, Ser. No. 62/687,402 filed on Jun. 20,2018, entitled, “Machine Learning Based Assignment of Service Levels ina Networked Storage System”, the disclosure of which is incorporatedherein by reference in its entirety.

TECHNICAL FIELD

The present disclosure relates to networked storage systems, andparticularly, to providing computing technology for defining customservice levels in a networked storage system using machine learning.

BACKGROUND

Various forms of storage systems are used today. These forms includedirect attached storage (DAS) network attached storage (NAS) systems,storage area networks (SANs), and others. Network storage systems arecommonly used for a variety of purposes, such as providing multipleusers with access to shared data, backing up data and others.

A storage system typically includes at least one computing systemexecuting a storage operating system for storing and retrieving data onbehalf of one or more client computing systems (“clients”). The storageoperating system stores and manages shared data containers in a set ofmass storage devices.

Cloud computing enables information technology infrastructure with bothcompute and storage resources to be consumed as a service. This hasmotivated traditional storage solution vendors, for example, NetAppInc., the assignee of this application to develop mechanisms fordelivering storage as a service as opposed to simply selling hardwareconfigurations and then letting customers determine service delivery.Because customers think of networked storage systems in terms ofservice, there is a need for computing technology to efficiently definecustom service levels based on the ability and usage of resources of adata center. Continuous efforts are being made to develop computingtechnology for efficiently managing a networked storage system providingcustomized service levels.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing features and other features will now be described withreference to the drawings of the various aspects. In the drawings, thesame components have the same reference numerals. The illustratedaspects are intended to illustrate, but not to limit the presentdisclosure. The drawings include the following Figures:

FIG. 1A shows an example of an operating environment for implementingthe various aspects of the present disclosure;

FIG. 1B shows a process flow for defining performance parameters usedfor discovering custom service levels of a networked storageenvironment, according to one aspect of the present disclosure;

FIG. 1C shows an example of a process for defining custom service levelsbased on machine learning, according to one aspect of the presentdisclosure;

FIGS. 1D-1L illustrate the process of FIG. 1C for defining customservice levels, according to one aspect of the present disclosure;

FIG. 2A shows an example of a clustered storage system, used accordingto one aspect of the present disclosure;

FIG. 2B shows an example of a storage system node, used according to oneaspect of the present disclosure;

FIG. 3 shows an example of a storage operating system, used according toone aspect of the present disclosure; and

FIG. 4 shows an example of a processing system, used according to oneaspect of the present disclosure.

DETAILED DESCRIPTION

As preliminary note, the terms “component”, “module”, “system,” and thelike as used herein are intended to refer to a computer-related entity,either software-executing general purpose processor, hardware, firmwareand a combination thereof. For example, a component may be, but is notlimited to being, a process running on a processor, a processor, anobject, an executable, a thread of execution, a program, and/or acomputer.

By way of illustration, both an application running on a server and theserver can be a component. One or more components may reside within aprocess and/or thread of execution, and a component may be localized onone computer and/or distributed between two or more computers. Also,these components can execute from various non-transitory, computerreadable media having various data structures stored thereon. Thecomponents may communicate via local and/or remote processes such as inaccordance with a signal having one or more data packets (e.g., datafrom one component interacting with another component in a local system,distributed system, and/or across a network such as the Internet withother systems via the signal).

Computer executable components can be stored, for example, onnon-transitory, computer readable media including, but not limited to,an ASIC (application specific integrated circuit), CD (compact disc),DVD (digital video disk), ROM (read only memory), floppy disk, harddisk, EEPROM (electrically erasable programmable read only memory),memory stick or any other storage device type, in accordance with theclaimed subject matter.

The system and techniques described herein are applicable and useful inthe cloud computing environment. Cloud computing means computingcapability that provides an abstraction between the computing resourceand its underlying technical architecture (e.g., servers, storage,networks), enabling convenient, on-demand network access to a sharedpool of configurable computing resources that can be rapidly provisionedand released with minimal management effort or service providerinteraction. The term “cloud” is intended to refer to the Internet andcloud computing allows shared resources, for example, software andinformation to be available, on-demand, like a public utility.

Typical cloud computing providers deliver common business applicationsonline which are accessed from another web service or software like aweb browser, while the software and data are stored remotely on servers.The cloud computing architecture uses a layered approach for providingapplication services. A first layer is an application layer that isexecuted at client computers. In this disclosure, the application allowsa client to access storage via a cloud.

After the application layer, is a cloud platform and cloudinfrastructure, followed by a “server” layer that includes hardware andcomputer software designed for cloud specific services. Detailsregarding these layers are not germane to the inventive aspects.

Conventional cloud computing technology today uses a static set ofservice levels for providing computing and storage services. Servicelevels are typically based on service level objectives (SLOs) thatdefine operating parameters for computing and storage services. Forexample, a SLO may define a certain latency level for reading andwriting data and/or executing a certain number of input/outputoperations per second (IOPS) in a networked storage system.

SLOs in conventional systems are typically defined by a rigid,menu-based, hierarchical levels, for example, Gold, Silver and Bronze,or Tier I, II and III service levels. A Gold service level provides acertain service level that may be higher than a Silver service level.This static approach fails to consider the actually operatingenvironment of a networked storage system and the overallperformance/utilization of the resources of the networked storage systemat any given time. This approach is also undesirable for a data center,where a user may want to transition an existing storage infrastructureinto a SLO based management framework for providing access to storageresources.

The conventional static approach is also undesirable for a cloud serviceprovider that may not be familiar with the underlying applications andstorage infrastructure. Therefore, the static, menu based approach,where service levels are defined by a standard menu are either toocoarse, requiring significant upgrades in storage system infrastructurethat will result in higher costs, or may cause significant degradationin service levels that may result in user dissatisfaction.

In the conventional storage environment, an existing storageinfrastructure transitions to SLO-based management manually by analyzingexisting workloads and then manually defining a service-level menu forproviding storage as a service. The manual approach can be tedious andinefficient because a data center uses a large number ofresources/storage objects for management with complex interactions, andalso deploys diverse workloads.

In one aspect, innovative computing technology is provided to enableusers to define custom service levels for providing storage and storageservices based on storage system resource capabilities. The innovativecomputing technology, implemented by a SLO module, addresses thechallenges of optimal SLO design and assignment of SLOs to storageworkloads with minimal manual intervention.

In one aspect, the SLO module retrieves performance data associated withdifferent resources of a storage system. The performance data may beretrieved from a management system, for example, NetApp OnCommand®Insight (OCI) (without derogation of any trademark rights) that isconnected to various storage controllers (or systems) and collectsperformance data from the storage systems. The SLO module appliesmachine-learning-based optimization algorithms for generating customservice level definitions and assignment of service levels to storagevolumes across different storage systems.

Before describing the details of the SLO module, the following providesan overview of a networked storage environment where the variousadaptive aspects of the present disclosure can be implemented.

System 100: FIG. 1A shows an example of a networked operatingenvironment 100 (also referred to as system 100), for implementing thevarious adaptive aspects of the present disclosure. In one aspect,system 100 may include a plurality of computing systems 104A-104N (mayalso be referred to and shown as server system (or server systems) 104or as host system (or host systems) 104) that may access one or morestorage systems 108 via a connection system 116 such as a local areanetwork (LAN), wide area network (WAN), the Internet and others. Theserver systems 104 may communicate with each other via connection system116, for example, for working collectively to provide data-accessservice to user consoles (or computing devices) 102A-102N (may bereferred to as user 102 or client system 102).

A cloud provider 140 may be used to provide storage and storage relatedservices (e.g. backup restore, cloning and other services) to clients.The cloud provider 140 may execute a SLO module 142 for customizingservice levels for storage system 108 in a data center. It is noteworthythat the SLO module 142 may be executed by server systems 104 or anyother computing device. The adaptive aspects disclosed herein are notlimited to any specific location for implementing the SLO module 142.

Server systems 104 may be computing devices configured to executeapplications 106A-106N (may be referred to as application 106 orapplications 106) over a variety of operating systems, including theUNIX® and Microsoft Windows® operating systems. Applications 106 mayutilize data services of storage system 108 to access, store, and managedata in a set of storage devices 110 that are described below in detail.Applications 106 may include a database program, an email program or anyother computer executable program.

Server systems 104 generally utilize file-based access protocols whenaccessing information (in the form of files and directories) over anetwork attached storage (NAS)-based network. Alternatively, serversystems 104 may use block-based access protocols, for example, the SmallComputer Systems Interface (SCSI) protocol encapsulated over TCP (iSCSI)and SCSI encapsulated over Fibre Channel (FCP) to access storage via astorage area network (SAN).

Server 104A executes a virtual machine environment 105, according to oneaspect. In the virtual machine environment 105, a physical resource istime-shared among a plurality of independently operating processorexecutable virtual machines (VMs). Each VM may function as aself-contained platform, running its own operating system (OS) andcomputer executable, application software. The computer executableinstructions running in a VM may be collectively referred to herein as“guest software”. In addition, resources available within the VM may bereferred to herein as “guest resources”.

The guest software expects to operate as if it were running on adedicated computer rather than in a VM. That is, the guest softwareexpects to control various events and have access to hardware resourceson a physical computing system (may also be referred to as a hostplatform) which may be referred to herein as “host hardware resources”.The host hardware resource may include one or more processors, resourcesresident on the processors (e.g., control registers, caches and others),memory (instructions residing in memory, e.g., descriptor tables), andother resources (e.g., input/output devices, host attached storage,network attached storage or other like storage) that reside in aphysical machine or are coupled to the host platform.

The virtual machine environment 105 includes a plurality of VMs113A-113N that execute a plurality of guest OS 115A-115N (may also bereferred to as guest OS 115) to share hardware resources 119. Asdescribed above, hardware resources 119 may include CPU, memory, I/Odevices, storage or any other hardware resource.

A virtual machine monitor (VMM) 121, for example, a processor executedhypervisor layer provided by VMWare Inc., Hyper-V layer provided byMicrosoft Corporation (without derogation of any third party trademarkrights) or any other virtualization layer type, presents and manages theplurality of guest OS 115. VMM 121 may include or interface with avirtualization layer (VIL) 117 that provides one or more virtualizedhardware resource 119 to each guest OS. For example, VIL 117 presentsphysical storage at storage devices 110 as virtual storage (for example,as a virtual hard drive (VHD)) to VMs 113A-113N. The VMs use the VHDs tostore information at storage devices 110.

In one aspect, VMM 121 is executed by server system 104A with VMs113A-113N. In another aspect, VMM 121 may be executed by a separatestand-alone computing system, often referred to as a hypervisor serveror VMM server and VMs 113A-113N are presented via another computersystem. It is noteworthy that various vendors provide virtualizationenvironments, for example, VMware Corporation, Microsoft Corporation(without derogation of any third party trademark rights) and others. Thegeneric virtualization environment described above with respect to FIG.1A may be customized depending on the virtual environment provider.

System 100 may also include a management system 118 for managing andconfiguring various elements of system 100. Management system 118 mayinclude one or more computing systems for retrieving storage system 108performance data and providing the same to SLO module 142. Managementsystem 118 may also execute or include a management application 138 thatprocesses performance data retrieved from the storage system 108, asdescribed below in detail. The performance data is provided to SLOmodule 142 for defining custom service levels.

In one aspect, storage system 108 is a shared storage system havingaccess to a set of mass storage devices 110 (may be referred to asstorage devices 110) within a storage subsystem 112. As an example,storage devices 110 may be a part of a storage array within the storagesub-system 112. Storage devices 110 are used by the storage system 108for storing information. The storage devices 110 may include writablestorage device media such as magnetic disks, video tape, optical, DVD,magnetic tape, non-volatile memory devices for example, self-encryptingdrives, flash memory devices and any other similar media adapted tostore information. The storage devices 110 may be organized as one ormore groups of Redundant Array of Independent (or Inexpensive) Disks(RAID). The various aspects disclosed herein are not limited to anyparticular storage device or storage device configuration.

In one aspect, to facilitate access to storage devices 110, a storageoperating system of storage system 108 “virtualizes” the storage spaceprovided by storage devices 110. The storage system 108 can present orexport data stored at storage devices 110 to server systems 104 and VMM121 as a storage volume or one or more qtree sub-volume units includinglogical unit numbers (LUNs). Each storage volume may be configured tostore data files (or data containers or data objects), scripts, wordprocessing documents, executable programs, and any other type ofstructured or unstructured data. From the perspective of the VMS/serversystems, each volume can appear to be a single disk drive. However, eachvolume can represent the storage space in one disk, an aggregate of someor all of the storage space in multiple disks, a RAID group, or anyother suitable set of storage space.

It is noteworthy that the term “disk” as used herein is intended to meanany storage device/space and not to limit the adaptive aspects to anyparticular type of storage device, for example, hard disks.

The storage system 108 may be used to store and manage information atstorage devices 110 based on a request generated by server system 104,management system 118, user 102 and/or a VM. The request may be based onfile-based access protocols, for example, the CIFS or the NFS protocol,over TCP/IP. Alternatively, the request may use block-based accessprotocols, for example, iSCSI or FCP.

As an example, in a typical mode of operation, server system 104 (or VMs113A-113N) transmits one or more input/output (I/O) commands, such as anNFS or CIFS request, over connection system 116 to the storage system108. Storage system 108 receives the request, issues one or more I/Ocommands to storage devices 110 to read or write the data on behalf ofthe server system 104, and issues an NFS or CIFS response containing therequested data over the connection system 116 to the respective serversystem 104.

The storage system 108 maintains a plurality of counters (not shown) totrack various performance parameters. For example, the storage system108 tracks latency for processing input/output (I/O) requests forclients for each storage volume. The storage system 108 may also trackthe number IOPS for each volume, the storage capacity that is used foreach volume and any rate of change of storage capacity utilization. Theperformance data maintained by the storage system 108 is provided to themanagement application 138. The performance data is also regularlyprovided to the SLO module 142 for defining custom service levels asdescribed below in detail.

In one aspect, storage system 108 may have a distributed architecture,for example, a cluster based system that may include a separate networkmodule and storage module, described below in detail with respect toFIG. 2A. Briefly, the network module is used to communicate with serversystems 104 and management system 118, while the storage module is usedto communicate with the storage devices 110.

SLO Module 142 Operations: FIG. 1B shows a process 150 for computingperformance parameters to define custom service levels for a data centerhaving a plurality of resources, including computing systems, storagedevices, network devices and other resources (e.g. system 100). Theprocess begins in block B152.

In block B154, performance parameters, P1-Pn, are defined for generatingcustomized service levels. The number of performance parameters mayvary. In one aspect, the SLO module 142 uses peak I/O density and peaklatency for each storage volume to define custom service levels suchthat storage volumes can be mapped to certain service levels and a“slack” (i.e. deviation from a defined service level) is minimal. Theprocess for determining the custom service levels is shown in FIG. 1Cand described below in detail.

In block B156, the performance data related to P1-Pn (.e.g. latency,IOPS, storage capacity utilization and other data) is retrieved from thestorage system 108. As an example, the performance data is retrievedfrom the management application 138. The performance parameters P1-Pnare computed by the SLO module 142 in block B158.

In one aspect, the SLO module 142 determines the peak latency and thepeak I/O (input/output) density for each workload. The term workload asused herein means a storage volume that is used for storing andretrieving data.

Peak latency is computed by the SLO module 142 based on volume levellatency (i.e. average latency for a storage volume) over a period oftime, for example, a month. The term latency as used herein means delayin processing I/O requests for a volume. The storage system 108maintains various counters to track the latency for each storage volume.The management application 138 retrieves the information from thestorage system 108 [Block B156]. The peak latency is computed as acertain percentile of the overall observation [Block B158]. For example,the 99th percentile or the 95th percentile of average latency may beused to represent peak latency. The maximum value is not considered as apeak because the maximum value may occur due to system aberrations.

Peak I/O density is computed by the SLO module 142 based on the numberof IOPS and a used capacity of a specific volume [B158]. In one aspect,the peak I/O density is computed over a period of time. The SLO module142 matches' time stamps of IOPS and used capacity measurements, and foreach time stamp, the I/O density is determined as a ratio of IOPS andthe used capacity for each volume. Once the I/O density for each timestamp is determined over a period of time (for example, 1 month), theSLO module 142 selects a certain percentile (99th or 95th) of theobservations to identify the peak I/O density for each volume.

FIG. 1C shows a machine learning, process flow 160 for defining servicelevels based on a data center's capability at any given time. The customservice levels optimize use of computing and networking resources of thedata center. The novel computing technology of the present disclosureprovides an interactive mechanism for defining service levels that thedata center can support. The term data center and networked storagesystem are used interchangeably throughout this specification.

In one aspect, the SLO module 142 uses a workload-centric approach fordefining different service levels so that workloads are accuratelymatched to appropriate service levels. In one aspect, the SLO module 142uses “slack” with respect to I/O density and latency to indicateworkload deviation from service level definitions. If the number ofservice levels is small, then manageability becomes easier but slack maybecome large, and the user unnecessarily pays more for storage services.On the other hand, a higher number of service levels results in lowerstorage infrastructure cost and increases overall manageability cost,since more service levels have to be supported by the data center.

The SLO module 142 provides an interactive computing tool in which auser can specify the number of service levels, slack; or both the slackand the number of service levels for defining service levels. Asdescribed below in detail, in the first case for specified number ofservice levels, volume performance parameters are clustered into anumber of specified service levels. In the second case where slack isspecified, the volume parameters are binned into several “bins”, eachhaving a size specified by slack parameters. In the third case for bothslack and number of service levels, the volume performance parametersare binned using the slack parameter and then the bins are furtherclustered into a specified number of service levels, as described below.

As an example, the SLO module 142 defines slack as a percentage of thepeak I/O density and the peak I/O latency for a service leveldefinition. The percentage is transformed into a logarithmic scale torepresent the I/O density and latency. Therefore, if the SLO definitionof I/O density is I and slack is p %, then in the log scale, the I/Odensity is represented as log2(I) and the deviation of the I/O densityfrom the definition level is defined as:s=log2(I(1+p/100)/I)=log2(1+p/100).

For 100% slack, s=1 and the I/O density in the logarithmic scale ispartitioned into bins of width=1. Similarly, if slack=300%, then s=2.The portioned bins of width=2 in the logarithmic scale. The sameconvention may be used to specify the slack for latency.

The SLO module 142 bins a volume parameter space along peak I/O densityand peak latency dimensions. In one aspect, workloads are represented bysingle points (peak I/O density, peak latency) in a space defined by thepeak I/O density and peak latency. The bin boundaries represent acorresponding SLO granule. A granule is data or information with certainvariations defined by granule boundaries.

The SLO module 142 determines granules by binning the volume parameterspace. Once the SLO granules are determined, they are clustereddepending on a number of specified service levels.

Referring back to FIG. 1C, process 160 begins in block B162. The SLOmodule 142 may receive slack parameters in block B164, the number ofdesired service levels in block B170 or both the slack parameter and thenumber of desired service levels in block B171. These parameters may bereceived via a CLI (command line interface), a GUI (graphical userinterface) or an API (application programming interface) executed by acomputing device (e.g. cloud provider 140).

When only the slack parameter is received, then in block B166, theperformance data is transformed, initial bins are created in block B168and the bin boundaries are adjusted in block B169, as described below indetail.

In one aspect, when only slack is specified, the SLO module 142 uses theslack values in the log scale for both peak latency and peak I/Odensity. The peak I/O density and peak latency are transformed in a logscale [B166]. For example, assume that the peak I/O density in the logscale ranges from I_(min) to I_(max). The SLO module 142 creates bins inthe log scale as (I_(max) to I_(max)−s) and (I_(max)−s to I_(min))[B168]. The bin boundaries are adjusted such that a minimum of the firstbin matches the minimum parameter value of the points falling withinthat bin [B169]. Similarly, the maximum of the second bin matches themaximum parameter values of the points falling in the second bin. Themaximum of the second bin is modified to I′_(max)<I_(max)−s. The binsare then partitioned by (I_(max)′ to I_(max)′−s) and (I_(max)′−s toI_(min)). The boundary of the newly created second bin is adjusted by(I_(max)′−s) such that the maximum parameter values falling in the newlycreated second bin (may be referred to as the third bin) matches themaximum. The process of block B169 continues until the minimum of thenewly created first bin is less than or equal to the I_(min). It isnoteworthy that process blocks B166-B169 are executed by a computingdevice for improving overall SLO management in a data center.

Similarly, the SLO module 142 partitions latency in the log scale byusing the specified slack. All possible combinations of I/O density andlatency of these bins in individual dimensions are used to constructtwo-dimensional granules. The total number of workloads present in eachgranule are determined, and if the number is less than a certainthreshold, the granule is ignored and may be referred to as an invalidgranule.

When only the number of desired service levels is specified [blockB170], then the initial bin boundaries are identified in block B172 andadjusted in block B174. In one aspect, this is executed by the SLOmodule 142 as follows:

As an example, for I/O density, DI (Delta I)=(I_(max)−I_(min)). The SLOmodule 142 computes dI=DI/n, where n is the number of specified servicelevels. In one aspect, DI is divided by a number larger than n. In suchcases, finer granules are obtained which result in lower slack.

For example, dI=DI/(5*n), then (I_(max) to I_(max)−dI) and (I_(max)−dIto I_(min)) are constructed as two bins [B172]. The bin boundary valuesare adjusted to match actual parameter values [B174]. Next the SLOmodule 142, computes the width of each bin. For example, let the widthsbe w₁ and w₂. If (5*n)>2, then the bin with the highest width isselected and dI is computed as =width/(the number of remaining bins+1).For example, if (5*n)=10 then dl=width/9. The bin is partitioned using(I_(max)′ to I_(max)′−dI) and (I_(max)′−dI to I_(min)′), where I_(max)′and I_(min)′ are boundaries of the bin.

The foregoing processor executable process continues until the entirespace is divided into, e.g., (5*n) bins. The same process is used forpeak latency, and for all possible combinations of peak latency and peakI/O density.

In one aspect, two-dimensional granules are constructed. The valid andinvalid granules are identified by comparing the number of volumespresent in each granule with a threshold value. For example, let amaximum of k workloads be allowed to be unmapped out of N workloads. TheSLO module 142 computes the percentage as k/N. If there are C granules(C=m*n in the first case and (5*n)² in the second case), then thethreshold is set as k/(C*N).

In block B176, the various workloads or volumes are grouped. Theworkloads are grouped by clustering valid granules. Each valid granuleis represented as one point for clustering represented by the center.The SLO module 142 does not consider the number of workloads within eachgranule, so that each granule gets the same importance. In other words,the data density of the granules is not considered, so that the servicelevel definitions with a higher number of workloads cannot distort thedefinitions with a smaller number of workloads.

In one aspect, the SLO module 142 uses a hierarchical clustering processto cluster the granule centers, using max(.) distance. Max(.) distancebetween two points in space A=[x_(y),y₁] and B=[x₂,y₂] is defined asDist(A,B)=max(|x₁−x₂|, |y₁−y₂|). Since this distance does not make thediagonal distance more than the distance in any individual dimension, byusing this distance the rectangular clusters are determined. The maximumcorner point of each cluster (max peak I/O density, max peak latency) isrepresented as the service-level definition of a corresponding cluster.If the number of service levels is specified, the granules are clusteredinto the specified number (n). If the number of service levels is notspecified, then each valid granule is considered as a service level.

When both slack and number of desired service levels are received by theSLO module 142, then in block B173 bins are generated and adjusted usingthe slack parameter, as described above. The bins are also clusteredinto specified number of service levels.

After the workloads are grouped, in block B178, a new workload is mappedto a defined service level based on the performance parameters for theworkload. The service-level definitions are represented as a set oftuples (peak I/O density, peak latency) and may be stored at a storagedevice.

To map any new workload, the SLO module 142 finds its peak I/O densityand peak latency, say (i,lt). Let the service level definitions be givenas (I₁,L₁), (I₂,L₂), . . . , (I_(k),L_(k)). The SLO module 142 finds allservice levels for which i<I and obtains k₁ for the service levels. Fromk₁, the service levels for which lt<L are determined. Thereafter, theSLO module 142 determines k₂ for such definitions. The difference (I−i)for all such k₂ possibilities are determined and mapped to the levelwhere the difference is minimal.

It is noteworthy that the service levels can be updated in block B180,as the data center continues to operate and machine learning is used tobetter understand the overall resource usage and performance of theresources of the data center.

FIGS. 1D-1L illustrate binning a parameter space, finding validgranules, and then grouping the granules, as described above withrespect to FIGS. 1B and 1C, respectively.

FIG. 1D shows the distribution of sample values. The shapes approximatesome distributions and the sizes show the number of samples. In thisexample, a sample represents the peak I/O density and the peak latencyof a volume that is represented as a single point in a space defined bylatency and I/O density. The geometric shapes show a simplified view ofthe distribution of latency and I/O density of different volumesconcentrated in that space.

FIG. 1E illustrates the bounds of I/O density (minimum peak I/O densityand maximum peak I/O density for all volumes). The latency and I/Odensity space is partitioned into 4×4 grids and partitioned into ¼^(th)and ¾^(th) segments.

FIG. 1F show that the boundaries of the ¼^(th) and ¾^(th) zones areadjusted.

As shown in FIG. 1G, the SLO module 142 determines that the ¾^(th) zoneis wider than the ¼^(th) zone after boundary adjustment. The ¾^(th) zoneis further partitioned into ⅓^(rd) and ⅔^(rd).

FIG. 1H shows that the bin boundaries are again adjusted.

FIG. 1I shows that four different bins are found by the SLO module 142.The width of the I/O density bins depends on data distribution, and arenot equal.

FIG. 1J shows the process of FIG. 1D-1H for the latency space to findfour bins for latency. Thus, the SLO module 142 obtains a 4×4 grid with16 granules, out of which 12 granules may have a considerable number ofsamples and one granule may have a very low number of samples. Thegranule with very low number of samples is ignored.

FIG. 1K shows that the 12 granules are clustered using max(.) distanceto obtain clusters of quadratic shapes. To cluster, the SLO module 142does not consider the number of samples present within the granules andonly the granule centroids are considered, as described above.

FIG. 1L shows that after the clusters are generated, the clusterboundaries are adjusted to find the maximum peak I/O density and themaximum peak latency. The pair of maximum peaks for every cluster isused to define the corresponding service level. As an example, FIG. 1Lshows three service levels.

In one aspect, the automatic discovery/definition of custom servicelevels can be implemented in different data centers. The SLO module 142may also use conventional, pre-set, service level definitions. Thisprovides options for data centers to map volumes by using pre-setdefinitions as well as to discover new definitions that are specific todata center capabilities.

When automatic discovery is used, it reduces the I/O density slack andreduces the number of unmapped volumes. This is useful for cloud serviceproviders that may want to have tighter control on the slack to providewell-defined service levels.

In conventional systems, SLO definitions have traditionally been treatedas a manual process based on the experience and expertise of a storagesystem expert. Conventional techniques do not discover SLO definitionsusing big data or machine-learning techniques, as described above. Thepresent disclosure provides a methodology of automatic discovery of SLOdefinitions that are customized for a data center's capability. Thisreduces dependency on manual expertise.

In one aspect, methods and systems for a networked storage system isprovided. One method includes transforming by a processor, performanceparameters associated with storage volumes of a storage system forrepresenting each storage volume as a data point in a parametric space;generating by the processor, a plurality of bins in the parametric spaceusing the transformed performance parameters; adjusting by theprocessor, bin boundaries for the plurality of bins for defining aplurality of service levels for the storage system based on theperformance parameters; and using the defined plurality of servicelevels for operating the storage system.

Clustered Storage System: FIG. 2A depicts an illustrative aspect of ashared, storage environment 200 where custom service levels can bedefined using the SLO module 142. The shared, storage environment 200includes the management system 118, the cloud provider 140, a pluralityof server systems 204.1-204.2 (similar to server systems 104), aclustered storage system 202 and at least one computer network 206communicably connecting the server systems 204.1-204.2 and the clusteredstorage system 202. The functionality of the cloud provider 140, the SLOmodule 142 and the management system 118 is described above in detail.

The clustered storage system 202 includes a plurality of nodes208.1-208.3, a cluster switching fabric 210, and a plurality of massstorage devices 212.1-212.3 (similar to 110, FIG. 1A). Each of theplurality of nodes 208.1-208.3 is configured to include a networkmodule, a storage module, and a management module, each of which can beimplemented as a separate processor executable or machine implementedmodule. Specifically, node 208.1 includes a network module 214.1, astorage module 216.1, and a management module 218.1, node 208.2 includesa network module 214.2, a storage module 216.2, and a management module218.2, and node 208.3 includes a network module 214.3, a storage module216.3, and a management module 218.3.

The network modules 214.1-214.3 include functionality that enables therespective nodes 208.1-208.3 to connect to one or more of the clientsystems 204.1-204.2 over the computer network 206, while the storagemodules 216.1-216.3 connect to one or more of the storage devices212.1-212.3.

The management modules 218.1-218.3 provide management functions for theclustered storage system 202. Accordingly, each of the plurality ofserver nodes 208.1-208.3 in the clustered storage server arrangementprovides the functionality of a storage server.

A switched virtualization layer including a plurality of virtualinterfaces (VIFs) 220 is provided below the interface between therespective network modules 214.1-214.3 and the client systems204.1-204.2, allowing storage 212.1-212.3 associated with the nodes208.1-208.3 to be presented to the client systems 204.1-204.2 as asingle shared storage pool. For example, the switched virtualizationlayer may implement a virtual interface architecture. FIG. 2A depictsonly the VIFs 220 at the interfaces to the network modules 214.1, 214.3for clarity of illustration.

The clustered storage system 202 can be organized into any suitablenumber of virtual servers (VServer or storage virtual machines (SVM))222A-222N, in which each virtual storage system represents a singlestorage system namespace with separate network access. Each virtualstorage system has a user domain and a security domain that are separatefrom the user and security domains of other virtual storage systems.Server systems 204 can access storage space via a VServer from any nodeof the clustered system 202.

Each of the nodes 208.1-208.3 may be defined as a computer adapted toprovide application services to one or more of the client systems204.1-204.2. In this context, a SVM is an instance of an applicationservice provided to a client system. The nodes 208.1-208.3 areinterconnected by the switching fabric 210, which, for example, may beembodied as a Gigabit Ethernet switch or any other switch type.

Although FIG. 2A depicts three network modules 214.1-214.3, the storagemodules 216.1-216.3, and the management modules 218.1-218.3, any othersuitable number of network modules, storage modules, and managementmodules may be provided. There may also be different numbers of networkmodules, storage modules, and/or management modules within the clusteredstorage system 202. For example, in alternative aspects, the clusteredstorage system 202 may include a plurality of network modules and aplurality of storage modules interconnected in a configuration that doesnot reflect a one-to-one correspondence between the network modules andstorage modules.

The server systems 204.1-204.2 of FIG. 2A may be implemented ascomputing devices configured to interact with the respective nodes208.1-208.3 in accordance with a client/server model of informationdelivery. In the presently disclosed aspect, the interaction between theserver systems 204.1-204.2 and the nodes 208.1-208.3 enable theprovision of network data storage services. Specifically, each serversystem 204.1, 204.2 may request the services of one of the respectivenodes 208.1, 208.2, 208.3, and that node may return the results of theservices requested by the client system by exchanging packets over thecomputer network 206, which may be wire-based, optical fiber, wireless,or any other suitable combination thereof. The server systems204.1-204.2 may issue packets according to file-based access protocols,such as the NFS or CIFS protocol, when accessing information in the formof files and directories.

In a typical mode of operation, one of the server systems 204.1-204.2transmits an NFS or CIFS request for data to one of the nodes208.1-208.3 within the clustered storage system 202, and the VIF 220associated with the respective node receives the client request. It isnoted that each VIF 220 within the clustered system 202 is a networkendpoint having an associated IP address. The server request typicallyincludes a file handle for a data file stored in a specified volume onat storage 212.1-212.3.

Storage System Node: FIG. 2B is a block diagram of a computing system224, according to one aspect. System 224 may be used by a stand-alonestorage system 108 and/or a storage system node operating within acluster based storage system described above with respect to FIG. 2A.

System 224 may include a plurality of processors 226A and 226B, a memory228, a network adapter 234, a cluster access adapter 238 (used for acluster environment), a storage adapter 240 and local storage 236interconnected by a system bus 232. The local storage 236 comprises oneor more storage devices, such as disks, utilized by the processors tolocally store configuration and other information.

The cluster access adapter 238 comprises a plurality of ports adapted tocouple system 224 to other nodes of a cluster as described above withrespect to FIG. 2A. In the illustrative aspect, Ethernet may be used asthe clustering protocol and interconnect media, although it will beapparent to those skilled in the art that other types of protocols andinterconnects may be utilized within the cluster architecture describedherein.

System 224 is illustratively embodied as a dual processor storage systemexe cuting a storage operating system 230 that preferably implements ahigh-level module, such as a file system, to logically organizeinformation as a hierarchical structure of named directories, files andspecial types of files called virtual disks (hereinafter generally“blocks”) on storage devices 110/212. However, it will be apparent tothose of ordinary skill in the art that the system 224 may alternativelycomprise a single or more than two processor systems.

Illustratively, one processor 226 executes the functions of a networkmodule on a node, while the other processor 226B executes the functionsof a storage module.

The memory 228 illustratively comprises storage locations that areaddressable by the processors and adapters for storing programmableinstructions and data structures. The processor and adapters may, inturn, comprise processing elements and/or logic circuitry configured toexecute the programmable instructions and manipulate the datastructures. It will be apparent to those skilled in the art that otherprocessing and memory means, including various computer readable media,may be used for storing and executing program instructions describedherein.

The storage operating system 230, portions of which is typicallyresident in memory and executed by the processing elements, functionallyorganizes the system 224 by, inter alia, invoking storage operations insupport of the storage service provided by storage system 108. Anexample of operating system 230 is the DATA ONTAP® (Registered trademarkof NetApp, Inc. operating system available from NetApp, Inc. thatimplements a Write Anywhere File Layout (WAFL® (Registered trademark ofNetApp, Inc.)) file system. However, it is expressly contemplated thatany appropriate storage operating system may be enhanced for use inaccordance with the inventive principles described herein. As such,where the term “ONTAP” is employed, it should be taken broadly to referto any storage operating system that is otherwise adaptable to theteachings of this invention.

The network adapter 234 comprises a plurality of ports adapted to couplethe system 224 to one or more server systems over point-to-point links,wide area networks, virtual private networks implemented over a publicnetwork (Internet) or a shared local area network. The network adapter234 thus may comprise the mechanical, electrical and signaling circuitryneeded to connect storage system 108 to the network. Illustratively, thecomputer network may be embodied as an Ethernet network or a FC network.

The storage adapter 240 cooperates with the storage operating system 230executing on the system 224 to access information requested by theserver systems 104 and management system 118 (FIG. 1A). The informationmay be stored on any type of attached array of writable storage devicemedia such as video tape, optical, DVD, magnetic tape, bubble memory,electronic random access memory, flash memory devices, micro-electromechanical and any other similar media adapted to store information,including data and parity information.

The storage adapter 240 comprises a plurality of ports havinginput/output (I/O) interface circuitry that couples to the disks over anI/O interconnect arrangement, such as a conventional high-performance,FC link topology.

In another aspect, instead of using a separate network and storageadapter, a converged adapter is used to process both network and storagetraffic.

Operating System: FIG. 3 illustrates a generic example of operatingsystem 230 executed by storage system 108, according to one aspect ofthe present disclosure. Storage operating system 230 interfaces with themanagement system 118 for providing performance data that can be used todefine custom service levels, described above in detail.

As an example, operating system 230 may include several modules, or“layers”. These layers include a file system manager 303 that keepstrack of a directory structure (hierarchy) of the data stored in storagedevices and manages read/write operations, i.e. executes read/writeoperations on disks in response to server system 104 requests.

Operating system 230 may also include a protocol layer 303 and anassociated network access layer 305, to allow system 200 to communicateover a network with other systems, such as server system 104 andmanagement system 118. Protocol layer 303 may implement one or more ofvarious higher-level network protocols, such as NFS, CIFS, HypertextTransfer Protocol (HTTP), TCP/IP and others, as described below.

Network access layer 305 may include one or more drivers, whichimplement one or more lower-level protocols to communicate over thenetwork, such as Ethernet. Interactions between server systems 104 andmass storage devices 110/212 are illustrated schematically as a path,which illustrates the flow of data through operating system 230.

The operating system 230 may also include a storage access layer 307 andan associated storage driver layer 309 to communicate with a storagedevice. The storage access layer 307 may implement a higher-level diskstorage protocol, such as RAID (redundant array of inexpensive disks),while the storage driver layer 309 may implement a lower-level storagedevice access protocol, such as FC or SCSI.

It should be noted that the software “path” through the operating systemlayers described above needed to perform data storage access for aclient request may alternatively be implemented in hardware. That is, inan alternate aspect of the disclosure, the storage access request datapath may be implemented as logic circuitry embodied within a fieldprogrammable gate array (FPGA) or an ASIC. This type of hardwareimplementation increases the performance of the file service provided bystorage system 108.

As used herein, the term “storage operating system” generally refers tothe computer-executable code operable on a computer to perform a storagefunction that manages data access and may implement data accesssemantics of a general purpose operating system. The storage operatingsystem can also be implemented as a microkernel, an application programoperating over a general-purpose operating system, such as UNIX® orWindows XP®, or as a general-purpose operating system with configurablefunctionality, which is configured for storage applications as describedherein.

In addition, it will be understood to those skilled in the art that theinvention described herein may apply to any type of special-purpose(e.g., file server, filer or storage serving appliance) orgeneral-purpose computer, including a standalone computer or portionthereof, embodied as or including a storage system. Moreover, theteachings of this disclosure can be adapted to a variety of storagesystem architectures including, but not limited to, a network-attachedstorage environment, a storage area network and a disk assemblydirectly-attached to a client or host computer. The term “storagesystem” should therefore be taken broadly to include such arrangementsin addition to any subsystems configured to perform a storage functionand associated with other equipment or systems.

Processing System: FIG. 4 is a high-level block diagram showing anexample of the architecture of a processing system, at a high level, inwhich executable instructions as described above can be implemented. Theprocessing system 400 can represent modules of management system 118,user console 102, server systems 104, cloud provider 140 and others.Note that certain standard and well-known components which are notgermane to the present invention are not shown in FIG. 4.

The processing system 400 includes one or more processors 402 and memory404, coupled to a bus system 405. The bus system 405 shown in FIG. 4 isan abstraction that represents any one or more separate physical busesand/or point-to-point connections, connected by appropriate bridges,adapters and/or controllers. The bus system 405, therefore, may include,for example, a system bus, a Peripheral Component Interconnect (PCI)bus, a HyperTransport or industry standard architecture (ISA) bus, asmall computer system interface (SCSI) bus, a universal serial bus(USB), or an Institute of Electrical and Electronics Engineers (IEEE)standard 1394 bus (sometimes referred to as “Firewire”).

The processors 402 are the central processing units (CPUs) of theprocessing system 400 and, thus, control its overall operation. Incertain aspects, the processors 402 accomplish this by executingprogrammable instructions stored in memory 404. A processor 402 may be,or may include, one or more programmable general-purpose orspecial-purpose microprocessors, digital signal processors (DSPs),programmable controllers, application specific integrated circuits(ASICs), programmable logic devices (PLDs), or the like, or acombination of such devices.

Memory 404 represents any form of random access memory (RAM), read-onlymemory (ROM), flash memory, or the like, or a combination of suchdevices. Memory 404 includes the main memory of the processing system400. Instructions 406 which implements techniques introduced above mayreside in and may be executed (by processors 402) from memory 404. Forexample, instructions 406 may include code used by the SLO module 142 aswell as instructions for executing the process blocks of FIGS. 1B and1C.

Also connected to the processors 402 through the bus system 405 are oneor more internal mass storage devices 410, and a network adapter 412.Internal mass storage devices 410 may be or may include any conventionalmedium for storing large volumes of data in a non-volatile manner, suchas one or more magnetic or optical based disks. The network adapter 412provides the processing system 400 with the ability to communicate withremote devices (e.g., storage servers) over a network and may be, forexample, an Ethernet adapter, a FC adapter, or the like. The processingsystem 400 also includes one or more input/output (I/O) devices 408coupled to the bus system 405. The I/O devices 408 may include, forexample, a display device, a keyboard, a mouse, etc.

Thus, methods and systems for dynamically defining service levels for adata center have been described. Note that references throughout thisspecification to “one aspect” or “an aspect” mean that a particularfeature, structure or characteristic described in connection with theaspect is included in at least one aspect of the present invention.Therefore, it is emphasized and should be appreciated that two or morereferences to “an aspect” or “one aspect” or “an alternative aspect” invarious portions of this specification are not necessarily all referringto the same aspect. Furthermore, the particular features, structures orcharacteristics being referred to may be combined as suitable in one ormore aspects of the present disclosure, as will be recognized by thoseof ordinary skill in the art.

While the present disclosure is described above with respect to what iscurrently considered its preferred aspects, it is to be understood thatthe disclosure is not limited to that described above. To the contrary,the disclosure is intended to cover various modifications and equivalentarrangements within the spirit and scope of the appended claims.

What is claimed is:
 1. A method, comprising; transforming by aprocessor, performance parameters associated with storage volumes of astorage system for representing each storage volume as a data point in aparametric space; generating by the processor, a plurality of bins inthe parametric space using the transformed performance parameters;adjusting by the processor, bin boundaries for the plurality of bins fordefining a plurality of service levels for the storage system based onthe performance parameters; and using the defined plurality of servicelevels for operating the storage system.
 2. The method of claim 1,wherein the transformed performance parameters include a peak latencyvalue for each storage volume.
 3. The method of claim 1, wherein thetransformed performance parameters include a peak Input/Output (I/O)density for each storage volume.
 4. The method of claim 1, wherein theplurality of service levels for the storage system are defined based ona specified number of service levels.
 5. The method of claim 1, whereinthe plurality of service levels for the storage system are defined basedon a specified deviation limit for the transformed performanceparameters.
 6. The method of claim 1, wherein the plurality of servicelevels for the storage system are defined based on a specified deviationlimit for the transformed performance parameters and a specified numberof service levels.
 7. The method of claim 1, wherein a new storagevolume of the storage system is mapped to one of the defined, pluralityof service levels.
 8. A non-transitory machine readable storage mediumhaving stored thereon instructions for performing a method, comprisingmachine executable code which when executed by at least one machine,causes the machine to: transform by a processor, performance parametersassociated with storage volumes of a storage system for representingeach storage volume as a data point in a parametric space; generate bythe processor, a plurality of bins in the parametric space using thetransformed performance parameters; adjust by the processor, binboundaries for the plurality of bins for defining a plurality of servicelevels for the storage system based on the performance parameters; anduse the defined plurality of service levels for operating the storagesystem.
 9. The non-transitory machine readable storage medium of claim8, wherein the transformed performance parameters include a peak latencyvalue for each storage volume.
 10. The non-transitory machine readablestorage medium of claim 8, wherein the transformed performanceparameters include a peak Input/Output (I/O) density for each storagevolume.
 11. The non-transitory machine readable storage medium of claim8, wherein the plurality of service levels for the storage system aredefined based on a specified number of service levels.
 12. Thenon-transitory machine readable storage medium of claim 8, wherein theplurality of service levels for the storage system are defined based ona specified deviation limit for the transformed performance parameters.13. The non-transitory machine readable storage medium of claim 8,wherein the plurality of service levels for the storage system aredefined based on a specified deviation limit for the transformedperformance parameters and a specified number of service levels.
 14. Thenon-transitory machine readable storage medium of claim 8, wherein a newstorage volume of the storage system is mapped to one of the defined,plurality of service levels.
 15. A system comprising: a memorycontaining machine readable medium comprising machine executable codehaving stored thereon instructions; and a processor module coupled tothe memory to execute the machine executable code to: transform by aprocessor, performance parameters associated with storage volumes of astorage system for representing each storage volume as a data point in aparametric space; generate by the processor, a plurality of bins in theparametric space using the transformed performance parameters; adjust bythe processor, bin boundaries for the plurality of bins for defining aplurality of service levels for the storage system based on theperformance parameters; and use the defined plurality of service levelsfor operating the storage system.
 16. The system of claim 15, whereinthe transformed performance parameters include a peak latency value foreach storage volume.
 17. The system of claim 15, wherein the pluralityof service levels for the storage system are defined based on aspecified number of service levels.
 18. The system of claim 15, whereinthe plurality of service levels for the storage system are defined basedon a specified deviation limit for the transformed performanceparameters.
 19. The system of claim 15, wherein the plurality of servicelevels for the storage system are defined based on a specified deviationlimit for the transformed performance parameters and a specified numberof service levels.
 20. The system of claim 15, wherein a new storagevolume of the storage system is mapped to one of the defined, pluralityof service levels.